Publish the service images as zstd rather than gzip - #369
Merged
Conversation
A pull was already a compressed transfer, so this is a better algorithm for the same job rather than compression where there was none. Measured on `agent-computer`, which is where the bytes are: 962 MB becomes 886 MB, and zstd inflates several times faster, which is worth more on 2 GB of Chromium than the 8% is. `force-compression=true` is the part that matters. Without it only our own thin layers change and the saving rounds to nothing, because almost every byte came from Playwright's base image. With it those layers are recompressed, which also means these images stop sharing layers with a gzip pull of the same base. Only the five component images. Reading a zstd layer needs a client that supports it, which Podman and current containerd do, and the installer ships Podman. `ghcr.io/copilotkit/openbot` is pulled by deployments running whatever they have, so it stays gzip. Verified against ghcr.io before merging rather than at the next release: both architectures pushed by digest with this exact exporter string, merged into one OCI index, layers reported as `application/vnd.oci.image.layer.v1.tar+zstd` including the base layer, and `docker pull --platform` succeeded for both.
davidmckayv
requested review from
MikeRyanDev,
guidovizoso and
tylerslaton
as code owners
September 4, 2026 18:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ahead of cutting 0.0.7, so the release publishes zstd images rather than needing a second release to get them.
What changes
One exporter string on the
component-imagesjob:oci-mediatypes=true,compression=zstd,force-compression=true.force-compressionis the part that matters. Without it only our own thin layers change and the saving rounds to nothing, because almost every byte came from Playwright's base image. With it those base layers are recompressed.The numbers, measured not estimated
agent-computer, arm64, pushed to a throwaway registry both ways:76 MB, 8%. The larger win is inflation speed, which zstd does several times faster than gzip, and that lands on 2 GB of Chromium on somebody's laptop during setup.
Only the component images
ghcr.io/copilotkit/openbotstays gzip. Reading a zstd layer needs a client that supports it: Podman and current containerd do, older Docker does not. The five component images are pulled by an installer that ships Podman; the single image is pulled by deployments running whatever they have.The cost, stated: these images no longer share layers with a gzip pull of the same base.
Verified against ghcr.io, not at the next release
A throwaway image pushed with this exact exporter string:
imagetools createmerged them intoapplication/vnd.oci.image.index.v1+jsonapplication/vnd.oci.image.layer.v1.tar+zstd, including the base layer, which is what provesforce-compressionreached itdocker pull --platform linux/amd64andlinux/arm64both succeeded from the merged tagzizmor --min-confidence lowclean, workflow YAML parses,format:checkclean.One cleanup I could not do
That probe left a private package,
openbot-zstd-probe, in the org's registry. My token lacksdelete:packages, so it needs removing by hand: https://github.com/orgs/CopilotKit/packages/container/package/openbot-zstd-probe